Gradient-based Regularization Parameter Selection for Problems with Non-smooth Penalty Functions

摘要

In high-dimensional and/or non-parametric regression problems, regularization(or penalization) is used to control model complexity and induce desiredstructure. Each penalty has a weight parameter that indicates how strongly thestructure corresponding to that penalty should be enforced. Typically theparameters are chosen to minimize the error on a separate validation set usinga simple grid search or a gradient-free optimization method. It is moreefficient to tune parameters if the gradient can be determined, but this isoften difficult for problems with non-smooth penalty functions. Here we showthat for many penalized regression problems, the validation loss is actuallysmooth almost-everywhere with respect to the penalty parameters. We cantherefore apply a modified gradient descent algorithm to tune parameters.Through simulation studies on example regression problems, we find thatincreasing the number of penalty parameters and tuning them using our methodcan decrease the generalization error.